Experiment I – 4 Keywords, 5 Methods
In Experiment I, I took 4 of my client’s most sought after keyphrases (from a popular 1-word query to a long-tail 4-word query) and measured rankings for a week using 5 different methods:
- Default – Standard, logged-in query
- PWS=0 – Adding the &pws=0 query parameter
- Logged out – Standard query, but logged out of Google
- Rank Tracker – Data from SEOmoz’s Rank Tracker tool
- GWT – Data from Google Webmaster Tools
Google Webmaster Tools data was only measured once, after the fact, using the “Last 7 Days” option. All queries were limited to web search on Google.com (US). The mean ranking for each keyword by method appears below:
Practically speaking, rankings for this particular set of keywords didn’t vary much across methods. Keyword 1 tends to bounce between the #1 and #2 spot, which the Logged out ranking showed, and there was some disagreement about Keyword 3, but the differences were mild at best. All methods correlated strongly with the default search (r = 0.97 to r = 1.00).
Experiment II – 20 Keywords, 6 Methods
Of course, this was one set of data and only 4 keywords/phrases, so I figured I should up the ante. I pulled the Top 20 search queries (by impressions) from Google Webmaster Tools and did a second round of one-day measurements. I also added a 6th method, “Caribou”. No, it’s not a secret codename – I took the laptop to Caribou Coffee to pull a new IP and tried a logged out search from there. Experiment II’s numbers turned out a bit more interesting:
This one takes a bit of explaining. Graphing 20 keywords x 6 methods is ugly at best, so Figure II shows the number of times each method’s ranking varied from default across 5 levels, from ±0 (same ranking) to ±4 spots. The PWS=0 and Logged out groups showed the least variation from default searches, with the Rank Tracker, GWT, and Caribou groups showing more variation (especially at ranking differences of 1-2 spots). Correlations ranged from a perfect 1.00 for the PWS=0 group down to r = 0.71 for Caribou and r = 0.69 for Rank Tracker.
What Does This Mean, Exactly?
I’m glad you asked. Of course, this doesn’t mean that the Rank Tracker and Caribou measures are unreliable. On the contrary, both correlated strongly (r = 0.90) with Google Webmaster Tools (GWT) rankings. What it does suggest is that, given enough keywords, rankings do vary a fair amount depending on how you measure them. There are a couple of general conclusions I think we can draw:
1. PWS=0 Is Unreliable
I’d welcome more data on the subject, but it looks like the “pws=0” parameter does little or nothing for many queries. If you’re using the de-personalization parameter and taking the result at face value, I’d strongly suggest you reconsider. It does appear that turning off personalization may affect some geo-targeted personalization, but the query parameter doesn’t make Google completely ignore your search history.
2. “Logged out” Probably Isn’t
Google’s announcement last year as much as admitted this – if you think being logged out will de-personalize your searches, think again. The open question is: just how much different is it? This data suggests that being logged out has very little impact on rankings, assuming that you’re on the same machine with the same IP. Move to a new machine/IP, and the difference is much more substantial.
3. Second Opinions Are Gold
There really is no gold standard. The rankings in Google Webmaster Tools are the closest we can get to being inside Fort Knox, but these numbers are completely opaque and many SEOs have reported occasional rankings that differ wildly from observed searches. If you rely on rankings as a primary metric, get a second opinion, preferably either a fully logged-out ranking on a browser/IP with no history, or by using a 3rd-party tool like SEOmoz’s Rank Tracker.
4. Skepticism Is Healthy
It always makes sense to check your facts, and search rankings are no different. Rankings vary – you can occasionally type the same query twice in a row and see two different results. Smart SEOs have already diversified, considering metrics like search traffic and conversions. On the other hand, even across these test cases, rankings don’t vary a huge amount. So, don’t panic, but as always, the key is not to put too much trust in any single number.
